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PREDICTION  OF  CHANGEOVER  PERFORMANCE: 
OPERATIONAL  TEST  (OT)  PARAMETERS  FROM 
DEVELOPMENTAL  TEST  (DT)  PARAMETERS 
VIA  META-ANALYSIS 


Donald  P.  Gaver 
Patricia  A.  Jacobs 
Arthur  Fries 


1.  Introduction 

This  paper  sketches  and  examines  some  analytical  statistical  concepts  and 
methodologies  that  should  usefully  inform  and  sharpen  the  process  of 
military  test  and  evaluation  decisionmaking.  The  concepts  generally  fall  into 
the  broad  category  of  combining  information  (Cl)  or  meta-analysis.  See  Gaver 
et  al.  (1992)  for  examples  and  references.  The  term  Cl  does  not  mean  blindly 
simplistic  and  uncritical  data  pooling  across  either  time  or  the  different 
systems  under  evaluation.  Cl  in  test  and  evaluation  would  encourage  and 
systematize  quantitative  descriptions  and  comparisons  between  systems' 
capabilities  and  limitations,  over  time  and  across  comparable  systems.  It  refers 
to  explicit  processes  whereby  judgments,  experience  and  expertise,  and  data 
from  previous  and  current  military  acquisitions,  are  systematically, 
transparently,  and  critically  brought  to  bear  on  data-taking  and  analysis  for 
either  a  particular  current  system  acquisition,  or  on  families  of  current  and 
future  projects.  It  quantifies  aspects  of  corporate  memory. 


The  formalized  Cl  process  illustrated  by  the  examples  we  provide  has  not 
yet  proceeded  far  in  practice  in  any  organized  way  because  useful  historical 
data  has  not  been  identified,  but  requirements  for  more  efficient  T&E 
decisionmaking  encourage  the  future  development  of  such  approaches. 


2.  Combining  Developmental  and  Operational  Testing  (DT  and  OT) 

Reliability  Data 

Developmental  Testing  (DT)  refers  to  the  testing  of  a  new  or  upgraded 
system  in  the  course  of  its  technical  engineering  development.  In  general, 
system  DT  is  conducted  by  technical  experts  attentive  to  demonstration  of  its 
engineering  performance  requirements.  Operational  Testing  (OT)  is 
conducted  later,  and  by  operational  military  personnel;  the  objective  is  to 
discover  how  the  system  is  likely  to  behave  in  field  operation  and  to  uncover 
faults  and  obstacles  to  such  operation.  Because  of  the  rigors  of  field  operation 
there  is  the  expectation  that  OT  failure  rates  are  likely  to  be  higher  than  those 
prevailing  in  DT.  We  sketch  analytical  models  that  can  represent  such 
behavior  in  a  quite  economical  or  parsimonious  way. 


Model  I:  A  Fixed  Changeover  Effect  Multiplier 

Suppose  that  6i  is  the  prior-to-changeover  failure  (or  event)  rate  for 
system  i,  and  coi  is  the  corresponding  post-changeover  rate;  i  =  1,  2,  ...,7. 
Assume  that  before  changeover  system  i  fails  in  accordance  with  a  Poisson 
process,  so,  over  operating  (exposure)  time  Xi,  system  i  fails  di  (di  =  0,  1, 2, ...) 

is  X 

times  with  probability  ^  — •  after  changeover  that  same  system  fails 

di'- 


( 

Wi  {wi  =  0,  1,  2, ...)  times  in  operating  time  y/  with  probability  — . 


Wj! 


this  independently  for  i  =  1,  2,  ...,/;  this  is  equivalent  to  assuming 
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exponentially  distributed  times  to /between  system  failures.  It  is  assumed  that 
the  data  initially  available  are  (di,  Xi,  Wi,  yi,  i  =  i,  2, ...).  Our  objective  is  to  use 
these  data  to  estimate  any  consistent  change  in  rates  {5i,  coi)  from  prior-  to 
post-changeover,  and  to  use  the  estimated  relationship  to  anticipate,  and 
strengthen  estimates  of,  the  post-changeover  rate  of  a  new  system.  The 
different  analysis  approaches  used  here  depend  on  different  ways  of 
characterizing  an  adjustment  factor,  ic,  k  is  first  taken  to  be  a  constant  in 
Model  I,  applicable  to  all  system  changeovers.  A  subsequent  setup.  Model  II, 
allows  the  data  to  indicate  the  constancy  of  the  relationship. 

Suppose  there  are  I  different  systems  for  which  both  DT  and  OT  data  are 
available.  Let  Dj  be  the  number  of  failures  experienced  by  system  i  during 
developmental  testing  (DT)  during  an  exposure  time  Xi.  Let  W,-  be  the 
number  of  failures  experienced  by  that  system  during  operational  testing  (OT) 
during  exposure  time  yi.  Model  I  assumes  that  {D,}  are  independent  Poisson 
random  variables  with  E[D,]  =  5iXi  and  {Wf}  are  independent  Poisson  random 
variables  with  E[Wi]  =  that  is,  /c  is  an  unknown  constant  in  this  model. 
The  log  likelihood  is,  up  to  multiplication  by  irrelevant  constants, 

I 

lnL  =  £{ 5, k; data)  =  ^  ) + dj  In  5,-  -  (K’^j-y,- )  +  Wi [in «• + In  ]  j .  (2.1) 

1=1 

d£ 

Setting  -—  =  0,  results  in 

OOf 


$  _  dj  +  ^i 
Xi+m 


(2.2) 


Setting 


9£ 

dK 


=  0, 
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(2.3) 


A.  -  __  yv - • 

i 

A  recursive  procedure  to  find  the  maximum  likelihood  estimates  works;  start 
with  5i{0)  =  difxi.  The  approximate  variance  of  it  can  be  obtained  from  Fisher 
information  or  by  bootstrapping;  details  are  omitted  here,  but  see  Gaver, 
Jacobs  and  Fries  (1997). 

An  important  use  of  the  estimate,  k,  is  to  project  DT  data  for  a  new  system 

into  the  post-changeover  OT  phase.  Suppose,  for  instance,  that  we  compute 

the  isolated  DT  rate  estimate  for  a  new  (the  I  +  1^9  system,  5/4.1=-^.  Then  a 

^1+1 

natural  point  estimate  for  the  failure  rate  during  OT  could  be 


a>l+i  =  k5i+i. 

Using  the  obvious  independence  and  asymptotic  likelihood  approximations 
(Fisher  information)  the  estimated  standard  error  (se)  of  dtj+i  can  be 
computed: 


=  }var[K]Var[5i^^]  +  V^k][e[Sm]]  +  Vflr[5j+i](E[ic])^ 


v2r[K] 


1  fd 


xi+i 


tl+l 

I  ^1+1. 


+  Var\K^ 


^  Y^j+i 

U/+i; 


(2.4) 


xi+lj 


(xf. 


This  can  in  turn  be  used  to  assign  approximate  standard  errors  to  predicted 
future  OT  performance,  such  as  the  probability  that  the  future  system  will 
exhibit  no/zero  failures  during  a  test  or  mission  time  xi+iim): 

P{Wj+i  =  0|wj+i,xj+i(m)}  = 


(2.5) 


Numerical  Examples 

Simulation  was  used  to  study  the  coverage  properties  of  various 
confidence  intervals  for  estimates  of  k  in  Model  1.  In  each  replication  20 
Poisson  random  numbers  are  generated  having  means  5ix\,  8\qx\q, 

8\Ky\, 5\QKy\Q,  where  k-  =  4  and  5i,xi,yi  appear  in  Table  1.  This  is  a 
simulated  version  of  raw  observational  data. 


TABLE  1 


System 

Number 

Si 

DT  Failure  Rate 

Xi 

DT  Test  Time 
(hours) 

Vi 

OT  Test  Time 
(hours) 

1 

0.0002 

20,000 

5000 

2 

0.0004 

10,000 

2500 

3 

0.0006 

6,666.67 

1666.67 

4 

0.0008 

5000 

1250 

5 

0.001 

4000 

1000 

6 

0.002 

2000 

500 

7 

0.004 

1000 

250 

8 

0.006 

666.67 

166.67 

9 

0.008 

500 

125 

10 

0.01 

400 

100 

For  each  replication  7  types  of  confidence  intervals  for  k  are  calculated. 
The  first  uses  the  MLE  estimate  of  k  and  the  asymptotic  normal  confidence 
limits  with  observed  Fisher  information.  The  next  three  procedures  use  2000 
bootstrap  replications  where  the  bootstrap  resampling  is  from  Poissons  with 
means  di,  xvi-  One  bootstrap  confidence  interval  procedure  uses  the 
percentiles  of  the  bootstrap  distribution.  Another  is  a  percentile-f  procedure 
with  the  observed  Fisher  information  of  the  bootstrap  sample  being  used  to 
estimate  the  standard  error;  (cf.  DiCiccio  and  Efron  (1996)).  The  third 
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procedure  uses  the  normal  confidence  interval  procedure  with  the  bootstrap 
standard  error.  The  last  three  confidence  intervals  are  also  obtained  by 
bootstrapping.  However,  in  this  case  the  2000  bootstrap  re-samples  are  drawn 
as  follows. 

1.  Obtain  re-samples  of  pre-changeover  (DT)  data  as  random  numbers  from 
the  Poisson  distribution  with  mean  SjXi. 

2.  Obtain  re-samples  from  post-changeover  (OT)  data  as  Poisson  samples 

A 

with  mean  Kdiyi 

A 

where  and  ic  are  the  original  parameter  estimates. 

The  three  confidence  interval  methods  are  the  percentile,  the  percentile-t, 
and  the  normal  confidence  interval  procedure  with  bootstrap  standard  error. 

Table  2  displays  the  results  of  the  simulations.  Displayed  are  the  number 
of  intervals  that  cover  the  true  k  =  4,  and  the  mean  and  standard  deviation  of 
the  width  of  the  intervals.  The  results  of  the  simulation  do  not  differ  by 
much  for  the  different  confidence  interval  procedures.  Thus,  for  practical 
purposes  the  convenient  asymptotic  confidence  interval  seems  adequate. 
Perhaps  by  luck  the  asymptotic  interval  not  only  covers  as  well  as  any,  but  is 
also  shorter  and  less  variable. 

Simulation  is  also  used  to  study  confidence  intervals  for  the  failure  rate  of 
the  post-changeover  OT  failure  rate  that  is  projected  from  DT  data  for  a  new 
system,  using  k.  In  each  replication,  data  is  simulated  using  the  model  with 
parameters  in  Table  1  with  «•=  4,  In  addition,  the  number  of  DT  failures  for  an 
Ifth  (the  new)  system  is  simulated  by  generating  dn  from  a  Poisson 
distribution  with  mean  0.004  x  1,000  conditioned  to  be  positive;  that  is,  5\\  = 
0.004  and  x\i  =  1000.  The  estimated  DT  failure  rate  of  the  11*^  system  = 


6 


A 

dnlxii  where  rfn  is  the  random  number  and  d)ii  =  K5ii  where  k  is  the 
estimate  obtained  from  the  data  generated  for  the  10  systems. 


TABLE  2 

Confidence  Interval  for  x:  Statistics 
100  replications 


Level 

80% 

90% 

95% 

Interval  Proc. 

Coverage 

Width 

Coverage 

Width 

Coverage 

Width 

Mean 

S.D. 

Mean 

S.D. 

Mean 

S.D. 

Asymptotic 

Normal 

85 

2.42 

0.59 

92 

3.11 

0.76 

95 

3.70 

0.90 

Bootstrap  I  (2 

000  Replications)  | 

Normal  B. 
SE 

84 

2.59 

0.67 

93 

3.32 

0.86 

95 

3.96 

1.02 

Percentile 

80 

2.49 

0.63 

91 

3.25 

0.81 

92 

3.94 

1.01 

t-Percentile 

82 

2.45 

0.60 

91 

3.17 

0.75 

92 

3.80 

0.90 

Bootstrap  II  (2000  Replications) 

Normal  B. 
SE 

86 

2.59 

0.68 

92 

3.32 

0.87 

95 

3.96 

1.08 

Percentile 

80 

2.48 

0.63 

91 

3.24 

0.83 

93 

3.94 

1.03 

t-Percentile 

82 

91 

3.17 

0.77 

93 

3.80 

0.91 

Bootstrap  I:  Dj(fo)  ~  Poisson  (dj)  ^■(^)  ~  Poisson  (w,) 

Bootstrap  II:  Dj(b)  ~  Poisson  ^i(^)  ~  Poisson  j 


For  each  replication  5  types  of  confidence  intervals  for  the  new  system's 
OT  failure  rate,  con,  are  calculated.  The  first  uses  the  MLE  estimate  of  k,  Sii, 
and  the  asymptotic  estimate  of  standard  error  and  normal  percentiles;  this 
confidence  interval  is  called  the  asymptotic  normal  interval.  The  next  two 
confidence  intervals  use  2000  bootstrap  replications,  where  the  bootstrap 
resampling  includes  the  additional  random  draw  of  duib)  from  a  Poisson 
distribution  with  mean  du,  the  number  of  DT  failures  for  the  system. 
The  fcth  bootstrap  estimate  of  (0\\  is  Wii(fc)  =  One  bootstrap 

confidence  interval  is  constructed  from  the  percentiles  of  the  bootstrap 
distribution  of  ©u-  The  second  procedure  uses  the  normal  confidence 
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interval  procedure  with  the  mean  and  standard  deviation  of  the  bootstrap 
estimates.  The  fourth  and  fifth  confidence  intervals  are  also  bootstrap 
confidence  intervals  but  with  the  bootstrap  samples  for  the  first  10  systems 

A 

being  drawn  as  follows:  di(b)  ~  Poisson  mean  S^Xi  and  zviib)  ~  Poisson  mean 

A 

icSiyi.  The  fourth  confidence  interval  uses  the  percentiles  of  the  bootstrap 
distribution  of  CDii  and  the  fifth  uses  the  mean  and  standard  deviation  of  the 
bootstrap  distribution  and  standard  normal  percentiles.  The  results  appear  in 
Table  3.  Displayed  are  the  number  of  intervals  (out  of  the  100)  that  cover  the 
true  value  of  g)ii  =  0.016  and  the  mean  and  standard  deviation  of  the  width  of 
the  intervals.  There  is  not  much  practical  difference  between  the  5  confidence 
intervals  procedures. 

TABLE  3 

Confidence  Intervals  for  OT  Failure  Rate  of  a  New  System 
K=  A,  =  0.004,  XII  =  1000,  =  0.016 

100  Replications 


Level 

80% 

90% 

95% 

Interval  Proc. 

Coverage 

Width 

Coverage 

Width 

Coverage 

Width 

Mean 

S.D. 

Mean 

S.D. 

Mean 

S.D. 

Asymptotic 

Normal 

82 

0.023 

0.009 

87 

0.030 

0.011 

91 

0.035 

_ 

0.013 

Bootstrap  I  (2 

000  Replications) 

Normal  B. 
SE 

84 

0.024 

0.009 

89 

0.031 

0.012 

91 

0.037 

0.014 

Percentile 

82 

0.023 

0.009 

89 

0.030 

0.012 

93 

0.036 

0.014 

Bootstrap  II  (2000  Replications) 

Normal  B. 
SE 

84 

0.024 

0.009 

89 

0.031 

0.012 

91 

0.037 

0.014 

Percentile 

81 

0.023 

0.009 

90 

0.030 

0.012 

93 

0.036 

0.014 

Bootstrap  I :  Dj{b)  ~  Poisson  mean  d^;  y^i{b)  ~  Poisson  mean  zv^,  i  =  1,...,10 


A  A 

Bootstrap  11:  Df  (fo)  ~  Poisson  mean  5,Xj ;  Wi{b)~  Poisson  mean  w/f  i  =  1, . . . ,  1 0 
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Model  II:  A  Variable/Adaptive  Changeover  Mtiltiplier 

This  next  model  generalizes  the  previous  setup  by  allowing  for  possible 
variability  in  the  DT-OT  multiplier,  K)  we  permit  the  training  data  sets, 
i  =  1,  2,  ...,  I,  a  chance  to  reveal  their  appropriate  K-variability:  each  training 
data  set  is  thought  of  as  having  its  own  jc-value,  each  a  sample  from  a 
population  with  mean  and  variance  to  be  estimated.  If  the  variability  of  this 
population  is  sizable  then  the  predictability  (and  usefulness)  of  the  relation  is 
questionable.  We  again  assume  {Df}  are  independent  Poisson  random 
variables  with  mean  SiXi,  and  we  let  {Wf},  number  of  failures  during  OT,  to 
be  independent  random  variables.  The  conditional  distribution  of  W/,  given 
Zf,  is  Poisson  with  mean  =  ^Zfyf  where  Zf  is  conveniently  taken  to  be  a 
gamma-distributed  random  variable  having  mean  ji  =  E[Zf]  =  ^  and  shape 

parameter  p  (its  scale  is  a),  so  each  system  has  its  own  individual  DT-OT 
multiplier;  these  may  be  close  to  a  mean  value  p  but  not  necessarily  tightly 
clustered  around  that  mean.  Note  that  this  is  not  the  same  as  a 
partially/ Bayesian  analysis  of  Model  I  with  k  an  unknown  constant  described 
by  a  gamma  prior.  In  the  present  model  {W/}  turn  out  to  be  independent 

negative-binomial  random  variables  that  depend  on  the  values  of  a  and  P, 
which  will  be  estimated  from  data.  The  value  of  ^V'flr[Zj]/£[Zj]  allows  an 

analyst  to  get  a  rough  idea  of  the  cohesiveness  of  his  data  sets. 

The  log-likelihood  is,  up  to  irrelevant  constants, 

^  =  lnL  = 


^  I  SiXi  +  dj  In  5i  +  In 
z=ll 


r(^+Wz) 

m 


+  p\na  +  wi  In  5/  -{p  +  Wj)ln[a  -i- 


.  (2.6) 
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Analysis  of  simulated  data  shows  that  procedures  to  maximize  the  full 
likelihood  rather  frequently  misbehave:  while  a  somewhat  reasonable 
estimate  of  the  gamma  mean,  =  is  usually  obtained,  the  tendency  is  for 

the  estimate  of  shape,  to  fly  towards  +<»,  so  the  variance  of  the  ic-population 
tends  to  be  badly  underestimated.  Such  misbehavior  of  likelihoods  has  been 
previously  noted  when  attempts  are  made  to  estimate  one  or  more  basic 
("interest")  parameters  in  the  presence  of  many  other  ("nuisance") 
parameters  (this  may  be  nature's  way  of  telling  an  analyst  to  slow  down);  see 
Cox  and  Reid  (1993),  (1987).  A  partially  Bayesian  way  of  addressing  the 
situation  is  to  treat  the  unknown  5s  as  random  and  integrate  them  out 
(marginalize).  Simulations  are  used  to  demonstrate  that  this  method  can  tend 
to  be  reasonably  reliable  -  much  more  so  than  is  the  full  likelihood  approach. 
A  similar  maneuver  has  been  employed  to  estimate  the  common  mean  of  a 
large  number  of  different-variance  normal  populations,  cf.  Bamdorff-Nielsen 
and  Cox  (1994). 

Assume  are  iid  with  a  Jeffery's  prior.  Then  to  marginalize  on  ^  carry 
out 


p{Di  =  di.W,  =  K),|Zi  =  q  =  1 


dj! 


Wj!  8i 


OCK^i 


di+TVi 


(2.7) 


Put  the  mean  {1  =  fi/ a.  The  integrated  likelihood  is  proportional  to 
L{ii,p-,data)  = 


I 

n 


n 


J  exp|-f — ^ — w  +  (wj-  +  jS  - 1)  In  u] 


(2.8) 


fir(/j)  UJyf 


[i+«f 


di+Wi 


■du. 
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After  several  algebraic  steps,  the  log-likelihood  is,  up  to  irrelevant  constants, 
£{lJ.,P;data)  =  '^l  pin  —  -lnr(j8)  +  )Slnxf -jSlny,- 

i=l  I  J 

(2.9) 

-i-ln  [expi-f  —  —  M  +  (zt;i  + /?-l) In M I - ^r— — du 

i  1  UJyi  ^  ‘  ^  ^  J 

Direct  numerical  integration  has  been  used  in  what  follows. 

Numerical  Examples 

Simulations  were  used  to  evaluate  the  above  procedures.  In  each 
simulation  replication  the  DT  rates,  5i,  i  =  1,  2,  I  were  generated 
independently  from  a  uniform  distribution  over  [0, 10].  The  times  Xi  =  yi=l 
for  all  simulations.  In  the  first  two  Cases,  (A)  and  (B),  the  {Zj}  values  were 


generated  from  a  gamma  distribution  having  mean  /i  =  4  and  shape 
parameter  =  1.  In  Cases  (C)  and  (D),  the  {Zf}  were  generated  from  a  gamma 
distribution  having  mean  /x  =  4  and  shape  parameter  P  =  4. 

For  each  data  set  generated,  the  mean  and  the  shape  parameters  of  the 
gamma  distribution  are  estimated  using  the  integrated  likelihood  (2.9),  and 
alternatively  the  method  of  moments.  The  numerical  integration  uses 
Simpson's  rule  with  up  to  10^^  order  difference  correction  for  a  step  size  h  = 
0.01  (cf.  Hamming  (1973))  over  [0,20]  as  implemented  in  A  Graphical 
Statistical  System,  AGSS. 


In  the  present  method  of  moments,  the  DT  failure  rate  of  the  system  is 
estimated  by  =  rfj7xj.  The  multiple  of  the  OT  failure  rate  for  the  system 
is  estimated  by  Xj  =  The  moment  estimate  of  the  mean  of  the  gamma 


1 


the  moment  estimate  of  the  shape  parameter  of  the 


gamma  is 


(2.10) 


(P-uf 


No  attempt  has  been  made  to  adjust  for  the  (Poisson)  variability  of  d,-  or  zvi  in 
the  above.  The  integrated  log  likelihood  is  searched  until  parameters  change 
by  less  than  0.01.  The  search  is  started  at  the  method  of  moments  estimates. 

Table  4  displays  the  results  of  a  simulation  experiment.  Each  simulation 
has  10  replications.  Each  replication  consists  of  I  systems.  The  mean  and  shape 
parameters  of  the  gamma  distribution  are  estimated  using  moments  and 
integrated  likelihood.  The  table  reports  the  mean  (and  standard  deviation)  of 
the  10  replications  of  the  estimates  of  the  mean  and  shape  parameter  of  the 
gamma  for  the  moment  estimators  and  the  integrated  likelihood  estimators 
for  each  case. 

The  results  displayed  in  Table  4  suggest  the  following.  Both  procedures 
estimate  the  mean  of  the  gamma  relatively  well;  they  are  both  biased  high, 

A 

with  the  bias  smaller  for  integrated  likelihood.  The  results  for  p  suggest  that 
the  method-of-moments  estimate  of  the  shape  parameter  can  be  biased  on  the 
low  side  for  larger  p.  The  variance  of  the  estimates  suggests  that  the  smaller 
the  variance  of  the  gamma  distribution  (larger  j5,  or  shape),  the  more  difficult 
it  is  for  either  of  the  procedures  to  accurately  estimate  the  shape  parameter  p. 

The  simulation  results  are  not  presented  as  at  all  exhaustive  or  definitive, 
but  as  suggestive  of  procedures  that  might  well  work  in  practice  (integrated 
likelihood,  but  also  simple  moments),  and  others  to  be  avoided  (full 
likelihood). 
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TABLE  4 

Estimates  of  Mean  and  Shape  Parameter  of  the  Gamma  Distribution 

10  replications 


1 

CasesXMethods 

Mean  (Std  Dev)  of 
Estimates 

Integrated  Likelihood 

Mean  (Std  Dev) 
of  Estimates 
Moments 

(A) 

/i  =  4,j3=l,/  =  10 

/Iml=  4.80  (1.75) 
j3ml  =  1-52  (0.50) 

/Im=  4.70  (2.45) 
Pm  =  1-08  (0.39) 

(B) 

^i  =  4:,p=l,I  =  20 

l^ML  -  4.33  (0.89) 
^ML=  1-66  (0.71) 

Pm  =4.37  0.32) 
Pm  =  1-22  (0.67) 

(C) 

H  =  4,P  =  4,I  =  1Q 

?ML  =4.55(1.47) 
i3ML=  5.38  (3.10) 

AIm=4.93  (1.82) 
Pm  =  1.81  (0.90) 

(D) 

fi  =  4,p  =  4,I  =  20 

4.34  (0.90) 

5.23  (4.28) 

^M=4.81  (1.22) 
Pm  =  1.38  (0.80) 

3.  Likelihood-Based  Pooling  of  Observations  from  Sensors  with  a 
Particular  Range-Dependent  Precision 

The  next  example  describes  the  form  of  an  estimate  of  a  target  item's  range 
from  observations  by  several  co-located  sensors  with  range-dependent 
precision.  Suppose  there  are  s  (s  >  1)  sensors  capable  of  detection  and  range 
determination  of  targets  at  various  ranges.  Here  is  a  model:  let 

Ri  =  range  estimate  of  sensor,  i  =  1,2, .. .,  s; 
if  r  is  the  true  range  of  target,  then  suppose  that  all  Rj  are 
independent  and  normal /Gaussian,  with 

E[Ri]  =  r 
Var[Ri]  =  afr^. 

The  objective  is  to  estimate  r,  using  all  information  available  in  the  above, 
which  means  use  the  fact  that  the  variance  also  depends  on  the  mean. 


13 


Treating  r  as  an  unknown  parameter  one  can  write  down  its  likelihood, 
given  observations 


It  is 


Rl  =  Xl,  R2  =  X2,  Rs  =  Xs. 


,1.. . 

IaX  t  Xl  /  ^2  /  •  •  •  /  Xg  )  11  t— — 

' - - - '  ■yjln  Gir 

Da  fa  1—1  * 


The  log-likelihood  is 


or,if  0=l/r 


Now 


£{r;data)  =  — slnr 

^i=l 


=  -oS-  12  ' 


2£-i 


i{G;data)  =  ^  ^ +  sln0. 


2^  fT? 

i=l 


(3.2) 


(3.3) 


(3.4) 


dl  _  1  2{xi0-l)xi  _  s 

~  2 
■i=l  <^i 


dd  2H  rr?  e 

i=\^i  1=1  ^ 

SO  setting  this  equal  to  zero  yields  a  quadratic  equation  for  ft 


f 


\  J 


(  „  ^ 


-s  =  0. 


V  ) 


(3.5) 


The  acceptable  solution  is 
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2=1^1 


+4s 


2S5 


consequently  the  point  estimate  that  uses  all  the  information 


--i- 

d~ 


s  a 

22\ 

i=l^i 


+4sy^ 


Under  certain  circumstances  (e.g.  <Tj  =  0(0.1))  the  second  term  inside  the 
radical  will  be  much  smaller  than  the  first;  neglecting  it  gives  the  weighted 


estimate 


f  =: 

~  s 


M _ _i=i _ _ 


It  is  surprising  that,  while  the  rightmost  formula  weights  as  the  inverse  of  the 
estimated/observed-range-calibrated  variance,  i.e.  a  natural 

surrogate  for  then,  instead  of  weighting  the  raw  observations  Xj,  it 

weights  their  fourth  power  with  the  normalizing  factor  necessarily  involving 
weighted  third  powers.  The  accuracy  of  these  estimates  should  be  compared  to 
the  simple  linearly-weighted  estimate  that  recognizes  true  range  dependence: 


1=1 

^L=—s 


M _ =  M _ 
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where  the  unknown  true  range  dependence  that  influences  the  variance 
cancels  in  this  (normal)  dispersion  model!  This  is  independent  of  the  form  of 
the  range  dependence,  g(r)  (here  g(r)  =  r^)  provided  it  is  the  same  for  all 
sensors.  Numerical  investigation  shows  that  the  linear  estimate  (3.9) 
performs  nearly  optimally,  and  is  certainly  highly  convenient  in  applications. 
Numerical  Illustration 

We  have  simulated  the  results  of  observing  a  target  by  five  different 
sensors  and  then  combining  the  results.  The  situation  and  results  appear  in 
Table  5,  and  the  figure  provides  further  insights. 

As  expected  the  mle  approach  (3.7)  generates  estimates  that  closely 
concentrate  around  the  true  mean  (r  =  1000);  the  standard  error  of  these 
estimates  is  smaller  than  those  of  the  other  two.  The  approximation  (3.8)  is  a 
bit  high  on  the  average  (dropping  a  positive  term  in  the  denominator  is  the 
reason).  The  properties  of  the  linear  calculation  (3.9)  are  gratifyingly  similar  to 
those  of  the  mle;  ease  of  calculation  is  welcome. 

TABLE  5 

Estimates  of  Range  Determination  by  Combination 
500  replications 

r  =  1000;  of  =  o|  =  0.09;  of  =o|  =05  =  0.05 


Estimate: 

MLE 

Approximation 

Linear 

Eq.  Number 

(3.7) 

(3.8) 

(3.9) 

Mean 

1006.4 

1068.8 

991.2 

Std.  Dev.  of  Estimates 

134.6 

143.1 

141.1 

Mean  Sq.  Error 

18118.6 

25169.2 

19939.9 

4.  Alternatives  to  the  Circular  Normal  Dispersion  Model:  "Robust  CEP" 

Experience  shows  that  in  projectile  (e.g.  missile,  gunfire)  testing  it  is  often 
the  case  that  some  individual  shots  deviate  from  aim  point  more  wildly  than 
described  by  the  customary  circular  normal  model.  One  way  of  providing  a 
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model  for  this  to  guide  data  analyses  and  perform  Cl  across  the  DT  to  OT 
changeover  is  to  stochastically  mix;  letting  p  =  l/o’^  be  a  precision  parameter, 
think  of  it  as  being  chosen  randomly,  perhaps  (not  necessarily)  from  shot  to 
shot,  and  then  presume  that  the  random  p  has  a  distribution  and  use  it  to 
remove  the  condition  on  precision. 

If  we  start  with  standard  circular  normal  dispersion,  conditional  on  p, 

p|k  >  rjp=  =  p|  =  ^  (4-1) 

then  when  the  condition  is  removed 


Gamma  Variation  of  the  Precision  Parameter 

Let  the  variability  in  the  precision,  p,  be  described  by  the  gamma  (a,  P) 
density 

dFp{x)  =  fp{x;a,p)dx  =  e~^  ^^^cdx;  (4.2) 

for  which  E[p]  =  p/a  =  p,  Var[p]  =  p/a^. 

Then 


P[R>r\a,P^  = 


f  1 

f 

(ryip/ayyn 


1h - p— 

2^  P 


.  (4.3) 


J 


The  CEP  satisfies  for  this  distribution  (which  has  a  long  or  fat  "Pareto  tail"). 


,  2 


1 

2 


(4.4) 


This  gives 
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The  CEP  approaches  ^21n2  1/p  =  1.177  as  ->  «>  if  p  =  1,  in  which  case  there 
is  no  shot  to  shot  variation  in  precision.  If  we  maintain  the  mean  of  the 
dispersion  distribution  at  p  =  1  and  reduce  fi  (increase  variance)  j3  =  l  (and 
p  =  1)  we  find  rcEP  =  =  1.414,  a  20%  increase;  as  P  decreases  further  the  CEP 

increases  indefinitely,  induced  by  the  great  shot-to-shot  variability. 

Parameter  Estimation  by  Likelihood 

Maximum  likelihood  estimation  of  the  parameters  of  the  miss-distance 
distribution  (4.3)  can  be  easily  done  when  all  miss  distances  of  a  series  of  n 
shots  are  recorded:  ri,  r2,  ...,  Tn-  That  is,  none  have  been  (prematurely)  deleted 
as  outliers.  The  density  function  of  a  miss  distance  is 

f  2  ■, 

/R(na,j3)=  1  +  yp--  rp  ;  (4.6) 

y  ^  P  J 

the  log-likelihood  function  is 


i{a,p}r)  =  -{p  +  l)^  In  l  +  +nlnp.  (4.7) 

f=lV  V  ^PJJ 

To  maximize,  differentiate  and  set  the  derivatives  equal  to  zero;  solve.  The 
equations  appear  as 


y  V - = 0 

dp  +  P 


(4.8,a) 


or 


1  _P  +  1  ly  r} 

p  2/5  n^^i  +  r^pl2p 


(4.8,b) 
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and 


dp 


=-E 


i=l 


In 


i+n 


2  P 


2p 


2/5'  h 


i+r'4 

*  2p 


=  0. 


(4.9,a) 


Because  of  (4.8,b)  the  second  term  allows 


‘  2/3  j  jS 


”z=l 


(4.9,b) 


A 

An  iterative  solution  of  (4.8,b)  and  (4.9,b)  is  promising:  start  from  p{0)  =  l, 
p(0)  =  2;3(0)f2’/«“)-lV(''MEDf  where  tmed  =  median  of  the  ordered  miss 


distances.  Use  of  Fisher  information  or  bootstrapping  will  furnish  standard 
errors  of  estimates. 

Modeling  DT-OT  Miss-Distance  Data  Combination 

A  convenient  approach  to  represent  the  difference  between  DT  and  OT 
miss  distances  in  the  current  context  is  to  utilize  a  proportional  hazard  or 
Lehmann  alternative  device.  If  are  respectively  location  errors  or 

shot  shot  deviations  for  DT  and  OT  we  put  in  (4.3)  for  the  DT  dispersion 


P{R^  >  r\a,p}  =  (p{Rd  >  r\a,p}f 


\Pk 


1  + 


{ry2)p-1/P 


The  corresponding  CEP  turns  out  to  be 


rcEE=M2yP^-l)^ 


(4.10) 


(4.11) 


so  here  k  turns  out  to  be  small  if  there  is  pronounced  degradation  of  precision 
in  OT  over  that  in  DT. 
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The  density  of  OT  miss  distances  is  seen  to  be 


^2  _  ^  '^+1) 


fv^{r;a,p,K)  =  \l+Yp-p\rpK 


(4.12) 


DT-OT  Data,  and  Parameter  Estimation 

Suppose  we  have  DT  miss-distance  data  from  system  i,  i  =  1,  2,  I, 
denoted  by  (DT)  =  Wy„/  =  1,  2, n/,  and  corresponding  OT  miss  distances, 

r^(OT)  =  Vij,  j  =  1,  2,  mi.  Under  Model  I  assumptions,  i.e.  a  fixed 
(unknown)  k,  the  likelihood  is 

^  f  U"  1  f  7,.  1 

rf/?  «  —  T7  TT  1 -lV  .  TT  i  j.  /■/! 


L(£,A^;d«ffl)=n  li  ^+-yPi-y  \Pi  -n  Pi^  ■  (4-^3) 

i=l  j=A  ^  PiJ  k=l\  ^  PiJ 
dropping  factors  independent  of  parameters.  Hence  the  log-likelihood  is 


I  f  U"  T  \  f  7  .  1 

p,p,K)data)  =  '^  -(A-  +  l)Xln  -(A-/c+l)£ln  l  +  -^p,-  — 

i=l  »=1  V  ^  PiJ  k=1  \  ^  Pi 


/=!  fc=l 


From  this. 


(4.14) 


4±iy — S - (felilg — m — +ivt^=o  (4.15^) 

2ft  ,ti  iH-^ft/ft  ^5  i+f  ft/ft  « 


1 

+  1 

Kj  +  mf 

2A- 

Y  _ ‘H 

^  “iV 

/=1  1  +  -2-. 


.+fe±2)^ 

9  ft. 


(4.15,b) 
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(4.16,a) 


dl 


rti 


f 


=  -Iln 

j=l  V 
m;  / 


Wi;_  1 


(A+ll  V 


l  +  _Xo._L  +v£:iJ_vy  _ -3. _ o- 

2  A  j  2A-  y.i  1  +  f  A-^ 


;t=i 


V  2  ^^A- 


+ 


(A^+1) 


m; 


m  k 

Because  of  (4.15,b)  the  second  term  allows 


2 

-1  0  Pi  ft. 


^ik 


2' Pi  Pi 


1 


A  ni  +  nii 

d£ 


“y  Pi 


IH 

u=^ 


1+—^ 

V  2  Av 


+'^XH 

it=i 


I  +  EiLB} 


V 


2  Pi 


I  mi 


r 


X-A'Iln 

z=l  A:=l 


1  +  — P:^ 

,  ^  Pi  y 


+  ^  =  0. 
K 


Thus, 


1  mi  f  —  \ 

-  -  ‘  ^kPi 


XftXH 

i  _  i=ifc=i 


1  +  - 


2  A 


’ly 


Z=1 


(4.16,b) 


(4.17) 


(4.18) 


An  iterative  solution  of  (4.15,b),  (4.16,b)  and  (4.18)  is  promising:  start  from 
A(0)  =  l,Pz(0)  =  ij  median^Tj  and  then  solve  for  ^(0)  using  (4.18). 


Notice  that  a  Model  II  version  of  the  present  setup  can  be  explicitly  carried 
out:  if  the  mixing  distribution  is  gamma,  as  before,  all  necessary  integrals 
come  out  in  closed  form  and  the  analysis  carried  forward.  This  work  must  be 
postponed  for  the  present. 

Numerical  Example 

Suppose  there  is  historical  data  on  5  similar  systems.  Each  system  has  20 
observations  during  DT,  and  20  observations  during  OT.  Odd-numbered 
systems  have  (p,  p)  =  (0.5,1).  Even-numbered  systems  have  (P,  p)  =  (1,2). 
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The  common  k  =  0.8.  Figures  2-4  present  results  from  500  replications  of  a 
simulation:  "data"  were  sampled  from  the  above-  model  and  the  parameters 
estimated.  Figure  2  presents  histograms  for  the  estimate  of  (p,  p)  for  the  odd 
numbered  systems,  and  Figure  3  presents  histograms  for  estimates  of  ip,  p)  for 
the  even  numbered  systems.  All  estimates  result  from  the  iterative  schemes 
of  (4.15,a)  -  (4.18).  Figure  4  presents  a  histogram  of  estimates  of  k.  The 
estimates  appear  well-behaved  in  that  their  histograms  cluster  well  around 
the  true  (here  known)  values,  doing  so  in  an  appropriately  normal  fashion. 

5.  Discussion 

The  present  paper  examines  a  selection  of  problem  types  typical  of  the 
testing  environment.  Emphasis  is  placed  on  the  issue  of  borrowing 
information  from  the  DT  period  to  strengthen  decisions  concerning  OT;  this 
can  be  useful  to  inform  the  decision  maker  of  the  advisability  of  the 
immediate  initiation  of  OT.  Some  attention  is  also  given  to  auxiliary 
information,  i.e.  with  respect  to  shot-to-shot  variability  change  with  range, 
and  the  occurrence  of  "fat-tailed"  outlier-prone  shot  dispersion  distributions. 
We  plan  to  further  address  such  problems  in  future. 
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